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ABSTRACT 

This paper argues that one. of the keys to an effective 
distance learning system is an effective monitoring and evaluation system. 

The evaluative process enables college faculty and administration to identify 
and. address problems distance learners are experiencing. Distance learning 
instructors at Albuquerque Technical Vocational Institute (New Mexico) 
developed a distance learning course evaluation survey for support personnel 
and instructors in order to address questions such as: Are course materials 
and objectives clear? Did the college provide the support students needed? 

Did course design meet the student's need for interaction with instructors? 
What course changes are necessary? A subcommittee consisting of faculty and 
staff members from several departments was charged with creating and 
administering a student survey and disseminating results. Distribution number 
is not reported, but the author comments that response rate was low. Findings 
indicated that: (1) 85% of distance learners were female; (2) 46% were 

between 31 and 49 years old; (3) 63% were White; and (4*) 77% worked full 
time; (5) 50% would not have taken a course if distance learning had not been 
available. ..The paper concludes with a description of problems encountered 
with the survey, and future implications. (NB) 
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"Who Are Our Learners? What Do They 
Expect From Us? An Evaluation Tale" 

Deb LaPointe 

Abstract 

One of the keys to an effective distance learning system is an effective monitoring and 
evaluation system (Moore, 1999). Through the evaluative process, the college faculty and 
leadership discover and address problems distance learners are experiencing. Albuquerque 
Technical Vocational Institute, a community college, (TVI) created a distance learning 
course evaluation survey and process to discover student perceptions about the distance 
learning courses with the goal of improving courses and support services. 

Introduction 

To what extent are distance learning courses meeting students' goals? =20 Are course 
materials and objectives clear? Do students perceive the college provided the support they 
needed at a distance? Did course design meet the students' need for interaction with 
instructors? What course changes are necessary? What are the students' reactions to the 
technology? Did the technology facilitate or hamper learning? Distance learning faculty and 
managers need answers to these questions to discover and address problems encountered 
by learners. Other faculty members— dubious, cautious, or hesitant observers of distance 
learning--are also interested in the answers. Likewise, college leadership wants to know 
answers to questions such as "Whom are we serving?" "How can we better retain distance 
learning students?" To answer these questions, distance learning instructors at Albuquerque 
Technical Vocational Institute, a community college, created a distance learning course 
evaluation survey that could provide formative evaluation for instructors and Student 
Service support personnel and a summative evaluation to aid in the college's future 
strategic planning. 

Accordingly, a distance learning faculty subcommittee was formed with faculty and staff 
members from several of the college's instructional departments, the Distance Learning 
Office, the Professional Development Center, and the office of Institutional Planning and 
Research. The subcommittee was charged to 1) create a survey, 2) establish a process for 
administering the survey, and 3) establish a process for disseminating information learned 
from the survey to instructors and other relevant departments and divisions. 

The subcommittee established the following criteria for an effective evaluation and process; 

The survey responses would be anonymous. 

The survey would be sent out the 12th week of the term to all 
students who registered for distance learning courses. 

The survey would reach students who dropped the course as well as 
those who completed the course. 

The surveys would be processed quickly with results returned 
immediately to faculty to review and incorporate into their classes for 
the following term. 

Mandatory questions regarding effectiveness of course content, learner 
support, technology questions, interaction across all courses and 
0 technologies would be asked by all faculty members. 

h 
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The survey would also be flexible and adaptable to meet individual 
faculty needs. The faculty members could select additional questions 
from an optional question database and as well as add their own 
additional questions specific to their courses or concerns. 

The subcommittee would start the pilot survey project with a paper- 
based survey as not all distance learning students had computer 
access. =20 However, the question database of mandatory and 
optional questions as well as the surveys would eventually move online 
in course Web sites. 

While individual instructors would receive their specific course survey 
results, anonymous global data would be gathered and shared with the 
Distance Learning Office, Student Services, the instructional deans, 
the Professional Development Center, and Institutional Planning and 
Research to improve processes. 

To fulfill its strategic planning function, the survey needed to also 
collect demographic data. 

Faculty and the Distance Learning Office would report back to students 
changes in courses and support services made based on the survey 
feedback. Constraints were numerous. To begin with, the instructional 
departments requested that no personnel other than the individual 
faculty member, the Distance Learning Office, and the instructional 
dean be able to track survey results to individual instructors. Second, 
the instructional departments used Scantrons to run surveys; 
however, the Office of Institutional Planning and Research created the 
surveys on asimilar but conflicting equipment. Third, not all students 
could be surveyed online. 

Fourth, not all faculty members had course Web sites in which to incorporate the online 
surveys. Finally, a lack of personnel who could create the online survance gathered from 
distance learning journals and distance learning Web sites. The subcommittee subsequently 
reviewed over 100 questions submitted, typed them out on individual sheets of paper, and 
grouped the questions according to categories that emerged. Categories that emerged from 
the process were 1) course content and learning activities, 2) student support, 3) 
technology, 4) interaction and feedback, and 5) open-ended questions. Duplicate questions 
were discarded. 

The categorized questions were then sent to each distance learning faculty member for the 
second round. Faculty members in this second round were asked to rate each question as 1) 
a required question all faculty members must ask on the survey, 2) an optional question 
faculty members could elect to add to individualize his or her survey, and 3) an irrelevant 
question to be thrown out of the question pool. 

Again faculty members independently rated the questions and returned their ratings. The 
ratings were tallied. Questions were printed out in order of faculty rankings; the questions 
that received the most votes as "essential" were listed first for the required and optional 
questions. =20 The questions rated irrelevant were discarded. Again instructors were asked 
to review and rank the top three essential questions in the required and optional categories. 




The subcommittee then went to work selecting and refining the top essential questions in 
each category. A final list of questions was sent to the faculty to allow them opportunity to 
express their opinions about the questions before the surveys were sent out to the students. 
The course evaluation survey now contains 31 essential questions. The subcommittee is 
currently revising the optional questions, working on the transition period as the online 
course evaluation survey created with Frontpage 2000 moves to the Web, creating the 
online database of required and optional questions, and building the Access database 
working behind the scenes. Faculty members will have to be trained to access the question 
database, create their course evaluation survey, and post it to their course Web sites. The 
Distance Learning Office is working on a verification process that the person submitting the 
course evaluation is a student, ensuring a student completes one evaluation survey per 
course, creating the process that works behind the survey to send survey results quickly to 
the relevant instructor as well as to the Distance Learning Office. During the current 
transition period, surveys will be completed online for courses with Web sites; written 
surveys will be mailed to students enrolled in courses that do not use the World Wide Web. 
=20 

Currently, the online completed survey results will be imported into an Access database. 

The responses on the paper-based surveys will be entered by hand on forms directly into 
the Access database. The faculty subcommittee has yet to develop the technology-specific 
questions. The college offers courses in six formats-audioconferencing, Internet courses, 
telecourses, dual courses (a combination of telecourse with required Internet access), 
correspondence, and CD-ROM. Faculty members are concerned with 1) differences in the 
way the various technologies facilitate or impede instruction and communication 2) 
frequency of technological problems, 3) ease of use, and 4) learner attitude toward 
technology. The faculty agree that the kinds of experiences learners have with different 
technologies and the impact of those experiences on their ability to learn influence their 
perceptions of delivery modes and satisfaction with a course as well as distance learning in 
general (Cragg, Andrusyszyn, & Humbert, 1999). The Distance Learning Office, too, hopes 
to hear from the learners about technological difficulties and issues that need to be 
addressed. 

Problems with the Survey 

TVI has administered the paper-based distance learning course evaluation survey on a pilot 
basis for three terms. A low return rate decreased the reliance faculty could place on the 
results, thereby, impacting the usefulness of the data received. Results have been low 
despite appeals from individual faculty members and an accompanying cover letter 
encouraging student participation and stressing the importance of student participation. 
Faculty members are hopeful that moving the survey to the Web within individual course 
Web sites will increase the return rate. 

What We Have Learned from the Survey 

Students who enroll in distance learning courses differ from the on-campus students. The 
distance learners are 85 percent female, 45.8 percent aged 31-49 years, 63.3 percent 
White; 77.5 percent work fulltime; 64 percent had previous college experience. More than 
half (55.9 percent) enrolled because on-campus courses conflicted with their work 
schedules, and 50 percent would not have taken a course if distance learning courses had 
been unavailable. About 56 percent noted their parents had not attended college; 67.2 
percent felt the time required to devote to a distance learning course was equivalent to an 
on-campus course. One third felt the amount of time required was more than that required 
by an on-campus course. It is unknown whether the students who felt a distance learning 
course required more time were first-time students. First-time students are often surprised 
by the amount of work in distance learning courses whereas experienced distance learners 
find the courses comparable in content and challenge to on-campus courses (Crane, 1985). 
1997 was the first year distance learning courses were offered at TVI, and few students had 
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taken distance learning courses prior to 1997. The majority reported student services were 
convenient to access and the courses helped them reach their educational goals. 

The 1997 survey indicated problems with the information given in the course schedule and 
the distance learning orientation. Students indicated that the TVI course schedule was the 
primary source of information regarding distance learning courses. However, only 71 
percent found the information available in the course schedule helpful in understanding the 
distance learning courses. Only 74 percent felt the course orientation helped them 
understand course content, teacher expectations, and the process for participating in the 
class. (In 1997, a televised distance learning orientation broadcast a panel of instructors 
discussing how the distance learning courses would work.) 

Based on this information, a distance learning section of the course schedule was created to 
contain a brief description of each course, testing requirements, number of times the 
student is required to come to campus, course Web sites, and contact numbers to give 
students the opportunity to learn as much information needed to make a decision regarding 
their potential success in a distance learning course. =20 Teachers began holding individual 
class orientation sessions, making orientation attendance mandatory, and showing students 
how to participate in the class during the orientation. Teachers now hold multiple sessions 
during both the first and second weeks of the term to orientate students who register late. 
The Distance Learning Web site contains information about each class, tips for succeeding 
as a distance learner, and contact information. The Distance Learning Office meets 
frequently with the advisors and counselors to share information that effects better 
advisement for students considering enrolling in distance learning courses. By 1999, 95 
percent of the students said the distance learning course information available before the 
class started provided clear information. 

Another concern discovered in 1977 was a large percentage of students who indicated that 
preproduced telecourses were somewhat disappointing. =20 Students said the telecourse 
study guide, videos, and text seemed disconnected and they were disappointed that the 
instructor or host in the telecourse videos was not their instructor. To address these 
concerns, instructors now stress how the textbook, telecourse study guide, and videos work 
together to create media-rich course content, and instructors suggest a logical approach to 
working through these materials. =20 Addressing the fact that students prefer to see their 
instructors, instructors are posting their photographs on their Web sites, incorporating audio 
files of themselves speaking on their Web sites, and some instructors are electing to 
produce their own telecourses. 

Interaction has been identified as one of the major constructs in distance education 
research (Mclsaac and Gunawardena, 1996). The TVI faculty, too, viewed interaction as an 
important factor in student motivation, intellectual commitment, class participation, 
personal development, satisfaction, and retention. Initially, in the 1997 survey, students 
said interaction with faculty was adequate while the faculty said they preferred more 
interaction with students. The 1997 survey indicated 83 percent of the students perceived 
adequate opportunities to interact with their instructors and timely instructor feedback. In 
the 1999 surveys, that percentage grew to 90 percent of the students agreed interaction 
with faculty members was adequate. 

While distance learners in 1997 indicated instructors provided timely feedback, results from 
the Summer 1999 surveys showed students were less satisfied with instructor feedback. An 
individual faculty member asked her class this open-ended question at an 1999 orientation: 
"What do you expect of me during this course?" The question produced 51 responses; =20 
over 60 percent of the student responses were to communicate in a timely manner (within 
24 hours) about adequacy of assignments and class participation and to provide clear help 
when needed. It appears as students acquire more experience in the immediacy of online 
environments, they expect feedback and responses from instructors within 24 hours. =20 
They also expect to receive more than an assignment with merely a letter grade written at 
the top. Because timely and encouraging feedback on assignments directly affect students' 




general sense of satisfaction with the course (Vrasidas and Mclsaac, 1999), instructors are 
now working to provide feedback through self-assessments, quizzes, and tutorials posted on 
their Web sites as well as inserting feedback in assignments submitted through e-mail as 
file attachments or in the body of e-mail messages. 

What is unknown at this time is the importance TVI distance learners perceive regarding 
interaction with their classmates. As instructors increasingly take advantage of the Web's 
collaborative tools and incorporate more learner-learner interaction in small group projects, 
debates, discussions, and learning communities to make courses more learner-centered, it 
will be important to hear learners' opinions about the value and importance of learner- 
learner interaction. 

Distance learning uses technology to facilitate this communication among faculty and 
learners that learners value. Anything that creates a barrier to or hinders that 
communication impacts learning. Accordingly, technological troubles are foremost concerns 
in instructors' minds. If an Internet connection goes down, communication and learning 
stop. =20 Technological support is crucial. Twenty percent of TVI students reported they did 
not get help when needed, and 20 percent related that the course was more difficult than 
expected due to the technology used to deliver the course. 

Technological support a difficult area for the college to address. =20 Currently the college's 
Computer Information Technology department provides support for the college's faculty and 
staff. No college department provides support for students; students must get help through 
their Internet service provider. Ultimately, however, faculty members serve as technology 
support for students encountering problems with their online courses and telecourse 
broadcasts. To help with this area of support, the Distance Learning Office decided it will 
post more specific information about technology needs and skills of distance learners as well 
as contact information for ISPs. As the faculty create more media-rich Web courses 
incorporating audio and videostreaming that require high-speed, high-bandwidth, the 
distance learning Web site and the course schedule will inform learners about ideal and 
adequate computer specifications and Internet connections needed to participate in the 
classes. 

The distance learning marketing materials were reviewed and revised with an eye for 
images or text that gave the appearance that distance learning was easy or less rigorous 
than on-campus courses. The survey results were used to better design advertising, 
recruiting, and admissions information that adequately represents the programs, 
requirements, and services available. 

The lessons learned from the course evaluation surveys have been communicated to the 
Professional Development Center, a unit that works with individual faculty members 
creating distance learning courses. The PDC is also developing a distance learning 
orientation course, incorporating ideas learned from the survey. 

The college's Computer Information Technology department recently used results and 
student responses to open-ended questions to strengthen its justification for a request to 
the college for two high-speed, high-bandwidth, redundant connections to the Internet. 

The survey results and actions taken by the faculty and college are posted on the Distance 
Learning Web site. The Spring 2000 online survey will include a confirmation page that 
informs the students that they may receive a copy of the results and actions taken by the 
faculty and the Distance Learning Office. 

Future of the Survey 

When the faculty subcommittee finishes revising the optional questions and the technology- 
related questions, the survey may be too long for posting online. Faculty subcommittee 
members may have to rethink and reduce the number of essential questions or redesign the 
appearance of the survey on the Web. When the students log on to the online survey and 
see screens of questions, they may log off immediately. Faculty and the college leadership 
will miss valuable input from the distance learners. 




7 



The distance learning course evaluation survey serves as one source of learner satisfaction 
with and feedback about distance learning courses. =20 Most faculty members regularly 
solicit feedback in chatroom discussions, e-mail, and in phone conversations. Faculty mail 
out or post plus/delta sheets and use other informal assessments to gather formative 
feedback throughout the course. 

While satisfaction with the distance learning courses appears healthy, the college is also 
concerned with ways of improving retention in distance learning courses. The college 
recently completed a phone retention survey of distance learning students, sampling 
students who successfully and nonsuccessfully completed distance learning students with a 
goal of learning how to better advise students, train faculty, market the courses, and design 
courses to improve retention. It appears from early scanning of the surveys, however, that 
factors outside the college courses-job and family responsibilities, illness and death, and 
financial problems— are the primary reasons students drop distance learning courses. That 
creates another challenge for distance learning faculty and leadership. 
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